Evaluation
76.6% of the respondents took part in an evaluation of the interview series. To gauge their skills, they were asked to complete several tasks on their phone. For some of the questions, there is a large difference between the treatment and control group! In all questions, the treatment group fared better, often by a surprising amount.
- 1Dramatic group difference in ability to connect to Wifi
- 2Big group difference in ability to point to library in Google Maps
- 3Slight group difference in ability to do a web search
- 4Large group difference in the ability to find and install an app
- 5Slight group and gender difference in showing storage
- 6Dramatic group difference in finding out if an app fits on phone
- 7Almost no difference in ability to change Facebook or Email password
- 8Recovering a lost password: medium group difference, large gender difference
- 9Adjusting browser settings for cutting costs is hard
- 10Extra: Password strength
- 11Evaluation Trivia Questions
Can the respondent conduct an activity with their smartphone, for the next questions?
Dramatic group difference in ability to connect to Wifi
The treatment group has a far higher succes rate when it comes to completing this task.
EV1. (Wi-Fi) My phone’s hotspot is turned on, and is called [your hotspot name here]. Show me how you can connect to it. (All respondents)
EV1. (Wi-Fi) My phone’s hotspot is turned on, and is called [your hotspot name here]. Show me how you can connect to it. (Treatment)
EV1. (Wi-Fi) My phone’s hotspot is turned on, and is called [your hotspot name here]. Show me how you can connect to it. (Control)
Notes from EV1, where answer is 'No problems completing or minor dificulties'
- None23
- She asked for password3
- Connected easily2
- Connected to WiFi easily2
- Completed the task easily1
- Completed the task well1
- Connected to WiFi easily.1
- Connecting to the WiFi without challenges1
- Coonected without issues1
- Could not connect to wifi1
- Could not find my wifi, later i showed her and she asked for password1
- Did not know the steps1
- Easy task1
- Easy task. Connected easily1
- Encountered no issue1
- He asked for a password1
- He asked for password1
- He can access wifi through flash share and xender without switching on the wifi1
- He could not see my wifi until I showed him1
- He first claimed he didn't know how to connect to wifi. He has always been asking for assistance when he came for workshops on how to get connected to the internet.1
Notes from EV1, where answer is 'Major dificulties'
- 1.Moses does not know the procedure of connecting through portable hotspot.2. Is it possible to share internet services through someone's phone.2
- Asia is not aware of the producer used to connect tethering and portable hotspot on her smartphone1
- Connected to WIFI and after that was not sure what next, had to be assisted.1
- Geoffrey does not know the procedure of connecting internet services through portable hotspot.1
- He does not understand the procedure of connecting through portable hotspot using setting apps on his smartphone1
- He first went to tethering and portable hotspot instead of wifi.On clicking on wifi, he didn't bother getting the wifi he was to get connected on, he said he was connected.On realising he could not access the internet though he claimed to be connected, he long pressed on wifi again and got my wifi and got connected after asking for my password.1
- How is it possible to share internet services from someone's phone1
- It has not responded well to the wifi and it was flactuating,I dont know why1
Notes from EV1, where answer is 'Unable to complete the task'
- 1.She clicked on sittings then clicked on wireless and network but was not able to identify tethering and portable hotspot.2
- Clicked on wifi but it did not get connected or bring the option of inputing a password.1
- Could not connect wifi completely, she has never used wifi1
- Doesn't know how to connect to WiFi. Was shown before but he forgot. Asked me to show him again1
- He "usually" uses wifi anytime he wants to use the internet, he switches on data bundles and wifi at a go. He touched on both data connection and wifi and said he was connected though he could not access the internet since he did not have an active data bundle.1
- He didn't know how to connect to wifi.1
- He doesn't know how to connect to wifi1
- He is used to open/unsecured wifi, he switched on wifi expected to get connected directly without inputing password.1
- He said he knew about wifi and how to get connected to it but he simply touched the wifi icon and claimed to be connected.1
- He touched on wifi but it did not give him options of inputing password but he told me, he was connected.1
EV2. (Google Maps) Ive been wondering where the nearest library is around here. Use your phone to direct me to the nearest one. (Treatment)
EV2. (Google Maps) Ive been wondering where the nearest library is around here. Use your phone to direct me to the nearest one. (Control)
Big group difference in ability to point to library in Google Maps
The treatment group has a far higher succes rate when it comes to completing this task.
Notes from EV2, where answer is 'No problems completing or minor dificulties'
- None25
- Completed task1
- Completed the task1
- Completed the task easily1
- Completed the task easily using Google Maps1
- Completed the task easily. No questions asked1
- Found it easily through maps1
- Google map,he used google map and it worked well1
- Google maps app, searched for "location of nearest library in Kilifi" he was brought the map but said he doesn't know how to interprete it, unless he had little knowledge about the area he is searching for.1
- Google maps-This is what she used in searching1
- Google mqp,this is what she used through to get the location of the library1
- He does not know which way to go about1
- He searched using Google Search not Google map1
- He used 'Nokia City Lens' which is an internal App for Nokia phone and gives directions. He easily completed the task1
- He used google maps app and searched for "Nearest library, malindi". He got the map searched for direction to get to the place.1
Notes from EV2, where answer is 'Major dificulties'
- Has never used maps before. He went ahead to google the National Library and couldn't get the clear directions, then agingAintree he remembered he has seen 'Maps' on his phone and that's when he got it at last. He asked if it's applicable on anywhere he wants to get direction to1
- He has never used Google map1
- He knows about google maps but can not use it but he tried open and try dialing but he had no idea of how and what to put down1
- He said he has never done it before but they had touched on it with Sterro.He went to gogle maps, searched for kenya national library but was directed to the Nairobi libraries and said that was it. On asking if he was sure that was the nearest to where we were that time he said tho§ were too far away.He then searched for knl, kilifi and got the one along Bofa road. But he does not know how to interprete the results.1
- He tried use google search but she dint manage because of her phone internet connection issue1
- Lilian is unable to identify google map app on her smartphone and use it to search locations1
- Moses cannot identify google maps apps on his smartphone1
- She cannot identify google maps app on her smartphone and does not know the procedure of searching a location on her smartphone.1
- She cannot identify google maps app on her smartphone and does not know the procedure of searching locations using google maps on her smartphone.1
- She cannot identify google maps app on her smartphone nor does she knows the producer of searching locations on her smartphone1
Notes from EV2, where answer is 'Unable to complete the task'
- Could not answer2
- According to her she said she would use GPRS but she did not have it on phone1
- Asked how to go about it. Was not sure of how to go about1
- Does not know which app to use to do the task1
- Doesn't know how to go about it. Didn't have an idea of an app on phone that could hep her in that.1
- Has never done it and doesn't know how to begin the process1
- Has never used Google Maps before and he asked to be shown.1
- He didn't know how to do the task, that he had never done the task and had no idea on how to complete it.1
- He didn't know how to get the information, and about the google Maps app.1
- He doesn't know how to get locations using his smartphone. Since they have wifi and the school computer lab/library, he wanted to connect to the school's wifi by logging out of my wifi-misunderstood the question.1
- He doesn't know how to go about it completely1
- He has never used Google map1
- He just clicked on google maps app and fansied on the maps displayed, he couldn't get to the requirements of the question.1
- He said he can't complete the task. I asked if he knew the function of google maps on his phone, he said he didn't have such an app while in real sense, he had it on his phone.1
- He used google search to find the location but it brought no direction (map )1
Slight group difference in ability to do a web search
In the treatment group, 8% more respondents abre able to do a simple web search without problems.
EV3. (Searching the web) Yesterday, my friend and I had an argument about which cities were the most populated in the world. Use your phone to find the top 3. (Treatment)
EV3. (Searching the web) Yesterday, my friend and I had an argument about which cities were the most populated in the world. Use your phone to find the top 3. (Control)
EV5. (Finding and Installing a trustworthy apps) Ive been researching a trip to China, and would like to learn some Mandarin Chinese before I go. Use your phone to find and install an app that I can trust to help me practice. (Treatment)
EV5. (Finding and Installing a trustworthy apps) Ive been researching a trip to China, and would like to learn some Mandarin Chinese before I go. Use your phone to find and install an app that I can trust to help me practice. (Control)
Large group difference in the ability to find and install an app
The treatment group has a far higher succes rate when it comes to completing this task.
Notes from EV5, where answer is 'No problems completing or minor dificulties'
- None37
- 1. Why do people trust apps downloaded from play store?1
- At first she searched from Google instead of Play store. She says she will look at the reviews to know if it's a good App1
- Completed task. She checked on the reviews and comments of other users1
- Did not actually download it but anaged to search through chinese language app through app store and said he would choose one among the list1
- Do you think all apps downloaded from play store are good.1
- Found an app easily from Play Store. She asked if one can save an app on a SD card1
- Go to opera mini and then google chinese language app1
- Got manadarin app through google search but did not download further1
- He did it so well1
- He used Google translator/Microsoft translator app and downloaded "virus" languages (the languages he prefers to translate to and fro-that one can download at their own pleasure), in his case he translated English to Mandarin chinese.1
- He used operamini to search for chinese language app but did not get or manage download any1
- He was ok with this activity, went to play store and located it.1
- His play store had issues but he managed later1
- How can be verify the app we have installed is trustworthy.1
Notes from EV5, where answer is 'Major dificulties'
- 1.Lydiah is unable to find,download and install an app on her smartphone 2. How do we know that an app is safe and free from virus1
- Could not answer1
- Could not tell how to get an app1
- He couldn't log into Play store cause he hadn't set up his gmail account, but he was later able to get the Apps1
- He first went to playstore. Searched from the given apps if he could find a related app but he didn't get any.He then searched for education but still didn't get any application related to what he was searching for.He then searched for mandarin Chinese and got a "learn mandarin" app that he is sure can help him out with the task.1
- He google searched and it said that he finds perfect transalater or download language app,so he tried further and it said he signs up as a translater but it dint work for him further than that1
- He said he Googles through opera mini1
- He said he knows apps are found on playstorebut he does not know how to get the right one1
- It was hard but he tried get some other apps on app store but did not download eg language app,duolingo but did not know which was the most reliable1
- None1
Notes from EV5, where answer is 'Unable to complete the task'
- "I fear even changing language settings in my phone and so I cant try anything even online because I think my phone may be compromised"This is what he says,because he things even a language app may alter her phone language settings1
- Asked me to show him how he can download Apps.1
- Asked to be shown1
- Can not tackle it1
- Cannot do anything ,says he does not know how to do it1
- Could not answer1
- Could not get the app1
- Could not get the app but through he went through playstore1
- Could not tell or do1
- Doesn't know how to download Applications1
- Doesn't know of an application used to download apps on her phone.She went to download app, searched for application for learning Chinese, but since she had not downloaded anything on the topic, she only got an advert on learning the language, and she claimed to have sussessfuly completed the task.1
- He doesn't know how to download an application on his phone. Those that he adds on his phone are what he is sent through flashshare(but recently he used 127mbs on google play store app).1
- He doesn't what application to download, if he knew what the app is a called, he could easily download it on playstore. He also doesn't have any other idea, how a smartphone can enable one learn a Chinese language because he first doesn't know their alphabet.1
Slight group and gender difference in showing storage
The treatment is a bit more succesful in pointing out the available space on their phone. Males are a bit more succesful than females for this task.
EV6. (Storage) My phone is always running out of space. Using your phone, show me how I would check if my own phone is almost full. (Treatment)
EV6. (Storage) My phone is always running out of space. Using your phone, show me how I would check if my own phone is almost full. (Treatment)
EV6. (Storage) My phone is always running out of space. Using your phone, show me how I would check if my own phone is almost full. (Treatment)
EV6. (Storage) My phone is always running out of space. Using your phone, show me how I would check if my own phone is almost full. (Treatment)
EV7. (Storage) Now, if I want to download an app, show me how I can tell if it will be too big to fit on my phone. (Treatment)
EV7. (Storage) Now, if I want to download an app, show me how I can tell if it will be too big to fit on my phone. (Control)
Dramatic group difference in finding out if an app fits on phone
The treatment group has a far higher succes rate when it comes to completing this task.
EV9. (Passwords) Without actually changing it, show me, or tell me the steps to change your password on facebook or email.
Almost no difference in ability to change Facebook or Email password
For these 2 different tasks, there was no large difference between groups or gender.
EV10. (Passwords 2) Let’s pretend you need to make a new facebook account. Show me how you would create a new password.
Recovering a lost password: medium group difference, large gender difference
EV11. Imagine you have lost your password for instagram/facebook/email. Can you tell us the steps to recover it? - All
EV11. Imagine you have lost your password for instagram/facebook/email. Can you tell us the steps to recover it? - Treatment
EV11. Imagine you have lost your password for instagram/facebook/email. Can you tell us the steps to recover it? - Male
EV11. Imagine you have lost your password for instagram/facebook/email. Can you tell us the steps to recover it? - Control
EV11. Imagine you have lost your password for instagram/facebook/email. Can you tell us the steps to recover it? - Female
Notes from EV11, where answer is 'No problems completing or minor dificulties'
- None43
- Also said it it the same as in going to forget password and then wait for new steps to followup to get a new one1
- Ask to give her recovery email, if one is able to provide the correct one, they'll send the recovery details to it.1
- Forget password1
- He knows how to go about it.1
- He says he normally backs up his emails on gmail, so it's easy to recover1
- He went to Facebook and typed a wrong password, then he clicked on 'forgot password'1
- He will just click on the 'Forgot Password' option and follow steps1
- If he doesn't remember the password, he would click on forget password, then followed the steps given towards getting a new password.1
- It will give me an option of 'Forgot Passoword' and it will send me a code to my number, then I create a new password. No questions asked or trouble1
- Knows how to retrieve passwords, even through Gmail account.1
- Logged out of her account, chose forget password and changed to a new password1
- No questions. Easy task, says she will just click on 'Forgot Password' and follow the steps1
- No trouble1
- On for example facebook account,there is always so place that asks one if they forgot,then that is the place to lead her1
Notes from EV11, where answer is 'Major dificulties'
- Geoffrey don't know the procedure of password recovery1
- He said he knows the process but wanted to go about the process on his active account. I suggested that he logs out before starting the process.He logged out then chose the option forgot password, then he input his username. He then chose the option of getting the reset code via phone number, it was sent to him and he successfully changed the password.1
- Said he sees some parts where one taps if they forget password but is not sure of what may be next.1
- She does not the know the procedure of recovering her password.1
- She says she always sees forget password sometimes and she thinks that is the place but not really sure1
- She says she will request for anotheron the forget password part and there another one 2ill come through some steps1
- She says there is a place she feels one can follow theough but she cant tell where particularly1
- Wilkister cannot read and write and this limits her chance for knowing the procedure of password recovery.1
Notes from EV11, where answer is 'Unable to complete the task'
- None2
- Asked to be shown1
- Could not answer1
- Does not know how to recover it1
- Does not know tge steps1
- Does not know the steps and have no idea1
- Doesn't know how and says unless he creates a new Facebook account1
- Has never changed any of his accounts and so he is not sure of the way through1
- He asked to be shown how to go about it1
- He didn't know how to begin the process.1
- He doesn't know how to1
- He doesn't know how to do it1
- He doesn't know how to renew a forgotten password. He knows there is the option of forgotten password but can't go through the steps to recovery.1
- He doesn't know how to retrieve a lost password1
- He has never done it and doesn't know how to try it out.1
EV12. (Browser settings) Show us one way to reduce data costs when you are using a browser (e.g. Opera Mini).
EV12. (Browser settings) Show us one way to reduce data costs when you are using a browser (e.g. Opera Mini).
Adjusting browser settings for cutting costs is hard
Showing a way to reduce data costs while using a browser proved to be a difficult tasks, as more than half of the respondents couldn't complete it.
Notes from EV12 where answer is 'No problems completing or minor dificulties'
- None12
- 1. Does opera mini have a button for data save?2
- How can we restrict background data usage2
- -Don't open many pages -Restrict bundles1
- By changing the network mode, changing its from 3G to 2G so that it may consume data slower.1
- By closing other apps not in use at the time he is using opera mini1
- By opening the site then switching off data bundles as he reads that page1
- By reducing some of the pages1
- By restricting background data1
- By restricting background data usage.1
- By restricting data use1
- By use of an app called operamax to control data bundles use1
- By use of Barground restrictions1
- By using combination of 2G and 3G only in the settings1
- Closing all the sites I dont use by minimising them and deleting to ensure tgey are not running1
Notes from EV12 where answer is 'Major dificulties'
- Can you explain to me how to restrict background data usage on my smartphone?1
- Could not answer1
- Guessed that he will makesure data is ever off when not needed1
- He said by using background data restrictions1
- He said he does not know,but maybe not having many apps that he does not use1
- How can we reduce data cost while using internet services on our phones.1
- How can we restrict apps which are not in use from consuming data while we are online1
- Is it possible to reduce data cost while online?1
- Is it possible to reduce data cost while using internet services1
- It took a lot of time to locate browser setting and to know what was expected of him.1
- Pamela does not know the procedure of restricting background data usage on her smartphone1
- Samson does not understand that apps such as opera mini which he use most of the time in his smartphone is cost effective in terms of data bundles1
- She would be specific to what she wanted so as not to make many searches in to different links.Also she ould ensure that her phone is always off mobile data unless she wants to use it .1
Notes from EV12 where answer is 'Unable to complete the task'
- Asked to be shown8
- None4
- She does not know2
- She doesn't know how to do it.2
- Asked to be shown how to reduce data costs1
- Cant tell1
- Cant tell any1
- Cant tell of any1
- Could not answer1
- Did not know and so she asked to be shown how to save data1
- Did not know of any1
- Does not have any idea1
- Does not k ow of any but he asked which ways they were1
- Does not know any method1
- Does not know of any1
Extra: Password strength
We ran the passwords that the respondents created for the question EV10. (Passwords 2) Let’s pretend you need to make a new facebook account. Show me how you would create a new password through a password strength checker. Most respondents pick a password that is to risky or easy to guess.
There are interesting difference between genders: females have a larger share of 'pretty good', and 'strong' passwords, but are more likely than men to pick a risky password.
Unfortunately there is no large difference between the treatment and control group here.
EV10 Added - Password strength score - Women
EV10 Added - Password strength score - Women
EV10 Added - Password strength score
Evaluation Trivia Questions
EV16.a. From whom do you get access to the internet?
| Respondent | EV16.b. What is an app? |
|---|---|
| Respondent | EV16.c.. What is a website? |
|---|---|
EV16.d. What is a browser?
EV16.e. Who can create apps, that you can download to your phone?
EV16.f. Who can create websites you visit?
| Respondent | EV16.d. What is a browser? |
|---|---|
| Respondent | EV16.e. Who can create apps, that you can download to your phone? |
|---|---|
| Respondent | EV16.f. Who can create websites you visit? |
|---|---|